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Abstract 

Few teachers receive adequate preparation to provide effective individualized instruction for 
children with intensive early writing needs. In this article, the authors describe an attempt to 
close this learning gap, by developing Data-Based Instruction-Tools, Learning, and Collaborative 
Support (DBI-TLC), a comprehensive professional development (PD) system that provides 
tools, learning opportunities, and ongoing collaborative supports for teachers to implement DBI 
in early writing. They describe the theoretical framework that has guided this work, the teacher 
population with whom they worked, their approach to assessing important teacher outcomes, 
and their development process. They highlight key findings that align with their theory of change, 


and discuss implications for further research and teacher preparation. 


Keywords 


general special education, quasi-experimental research, research methods, writing instruction, 
response to intervention, assessment, curriculum-based measurement 


Recent national efforts (e.g., National Center 
for Intensive Intervention; intensiveinterven- 
tion.org) have aimed to improve teachers’ 
individualization of instruction. These efforts 
have focused primarily on reading and math, 
with much less attention on writing. Yet, 
learning to write is critical to students’ literacy 
development (Biancarosa & Snow, 2004; Gra- 
ham & Hebert, 2010), their overall academic 
performance in school (Shanahan, 2004), and 
their future postsecondary and employment 
opportunities (Graham & Perin, 2007). Stu- 
dents with disabilities are particularly at risk 
for writing failure; for example, in the 2011 
National Assessment of Educational Progress 
(NAEP), only 5% of eighth graders with 
learning disabilities (LD) reached proficiency 
in writing; 60% were below basic levels 
(National Center for Education Statistics, 


2012). Furthermore, these statistics have not 
improved since the 2007 NAEP results. 

To address students’ writing development 
needs, particularly for those who experience 
significant difficulties, teachers require 
knowledge, skills, and tools for effective writ- 
ing instruction and intervention. Yet, there is 
little evidence that teachers are prepared to 
deliver effective writing instruction. In a 
national survey of elementary teachers, more 
than half cited their teacher education pro- 
grams as poor to merely adequate in preparing 
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them to deliver effective writing instruction 
(Cutler & Graham, 2008). Furthermore, in a 
study of the relation between teacher knowl- 
edge and time allocation in literacy instruc- 
tion (Spear-Swerling & Zibulsky, 2014), 
many general and special education teach- 
ers—particularly those in primary grades— 
allocated little to no time to writing assessment 
or basic writing skills instruction, despite evi- 
dence that such instruction is foundational for 
later writing proficiency (e.g., Berninger, 
Nielsen, Abbott, Wijsman, & Raskind, 2008). 

To address these gaps in teacher prepara- 
tion, our research team was funded a devel- 
opment and innovation project by the 
Institute of Education Sciences (IES). These 
development and innovation—or Goal 2— 
projects provide the opportunity to explore 
research questions aligned to three phases: 
development, feasibility testing, and pilot 
testing. The purpose of our Goal 2 project 
was to develop a professional development 
(PD) system that provides tools, learning 
opportunities, and ongoing collaborative 
supports for teachers to effectively imple- 
ment and individualize early writing instruc- 
tion using Data-Based Instruction (DBI). 
DBI is a_ hypothesis-driven, empirical 
approach to individualizing instruction 
(Deno & Mirkin, 1977) that entails a system- 
atic, ongoing cycle of assessment and inter- 
vention delivered in addition to (or instead 
of) general core instruction (Danielson & 
Rosenquist, 2014; see Figure 2 for basic 
steps in the DBI process). 

For the purposes of this project, we charac- 
terize DBI as a Tier 3 intervention (in a 
Response to Intervention model) or as an 
intervention delivered to students who receive 
special education services. As part of this 
project, we developed and revised DBI-TLC, 
which includes the following components: 


a. Tools, which include Curriculum- 
Based Measures (CBM) for monitor- 
ing student progress in writing (see 
McMaster, Ritchey, & Lembke, 2011, 
for a review), research-based interven- 
tion lessons and materials (see McMas- 
ter, Kunkel, Shin, Jung, & Lembke, in 


press, for a review), and decision-mak- 
ing tools; 

b. Learning modules (a series of face-to- 
face workshops designed to train teach- 
ers to use the tools and implement all 
components of DBI with fidelity); and 

c. Collaborative support (ongoing coach- 
ing to support teachers’ implementation 
of DBI). 


Across 3 years, we tested DBI-TLC 
through a series of iterative phases that 
included development, feasibility testing, and 
finally, a pilot study that entailed a small ran- 
domized control trial. The focus of this article 
is the process and teacher outcomes that were 
part of each phase of iterative development 
and refinement. Below, we describe our theo- 
retical framework, the teacher population that 
we aimed to support, how we assessed critical 
teacher outcomes, our process of develop- 
ment and refinement, and key findings. We 
end with implications for future research and 
teacher preparation and PD. 


Theoretical Framework 


Desimone (2009) proposed that researchers 
studying PD should base their work on a com- 
mon conceptual framework that would “ele- 
vate the quality of professional development 
studies and subsequently the general under- 
standing of how best to shape and implement 
teacher learning opportunities for the maxi- 
mum benefit of both teachers and students” 
(p. 181). This framework assumes that effec- 
tive PD should lead to increased teacher 
knowledge, skills, and attitudes, leading to 
positive changes in practice and improved 
student outcomes. Desimone proposed sev- 
eral features that effective PD should include. 
We incorporated these features into DBI-TLC 
in the following ways: (a) focus on content, by 
emphasizing knowledge and skills needed to 
implement each DBI step; (b) active learning 
opportunities over an extended duration, by 
providing multiple chances to view models, 
practice, and apply content with feedback dur- 
ing workshops and in the classroom; (c) 
coherence, by ensuring that DBI components 
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Data-Based Instruction for Beginning Writers 


Step 1: Establish present level of writing performance 


Step 2: Set ambitious long-term goal | 


v 


Step 3: Implement high-quality instruction 
with fidelity 


v 


Step 4: Monitor progress toward the goal 


¥ 


Step 5: Use decision rules to evaluate 


instructional effectiveness and student 
progress 


¥ 


Step 6: Generate hypotheses about 
student progress to individualize 


instruction 


Step 7: Make an instructional change 
based on the hypothesis chosen in Step 6 


v 


Step 8: Repeat steps 4-7 as necessary 


Figure I. Theory of change. 


align with theory and core academic stan- 
dards, and are integrated into existing instruc- 
tional routines; and (d) collective participation, 
by ensuring that teachers have frequent oppor- 
tunities to collaborate with coaches and peers. 
Our goal was to provide teachers with sup- 
port to implement DBI with their students, 
including PD in critical areas, ongoing coach- 
ing visits, and supported practice of activities 
that would benefit implementation fidelity. 
Our PD and coaching was focused both on the 
DBI elements and also on implementation of 
writing instruction. As illustrated in Figure 1, 
our work was guided by a theory of change 
that aligns with Desimone’s (2009) frame- 
work, and that relies on five assumptions: 


1. An important goal is to improve stu- 
dent outcomes in early writing. 

2. Given that no single approach will 
work for all students (Deno, 1990), 
teachers must make timely and appro- 
priate instructional decisions for stu- 
dents with significant needs. 

3. To make timely and appropriate 
instructional decisions, teachers need 
a strong framework—such as DBI— 
and must implement it with fidelity 
(Stecker, Fuchs, & Fuchs, 2005). 

4. To implement a practice such as DBI 
with fidelity, teachers require knowl- 
edge and skills of that practice (e.g., 
Cunningham, Perry, Stanovich, & 
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4a 
Increased 
Teacher 
Knowledge 
and Skills 


5 DBI-TLC: 
Tools, 
Learning, 


Collaborative 
Support Increased 
Teacher 


Efficacy 


3 Teachers’ 
use of DBI 


with fidelity 


2 Timely and 1 

appropriate Improved 
individualization student 

of instruction outcomes 


Figure 2. Steps in the DBI process. 
Note. DBI = Data-Based Instruction. 


Stanovich, 2004; Spear-Swerling, 
2009). Teachers also require self-effi- 
cacy, or “confidence that they can per- 
form the actions that lead to student 
learning” (Graham, Harris, Fink, & 
MacArthur, 2001, p. 178). 

5. We can increase teacher knowledge, 
skills, and self-efficacy through DBI- 
TLC. By providing in-depth PD in the 
theoretical and empirical underpin- 
nings, as well as classroom applica- 
tions, of DBI, we can increase teacher 
knowledge and skills. By providing 
ongoing support that allows teachers 
to experience success in applying new 
knowledge and skills to their instruc- 
tional practices, we can increase 
teacher efficacy related to DBI and 
early writing. 


Teacher Population Studied 


In our project, we aimed to support teachers 
who provided direct instructional services to 
children in Grades | to 3 with intensive early 
writing needs, including special education 
teachers, Title I teachers, and teachers sup- 
porting English Learners. We also aimed to 
support teachers in different types of settings 
serving diverse student populations. Thus, 
teachers who participated in this project were 
located in two demographically diverse dis- 
tricts in two states. The districts were in a 
small city and a large urban area, with K-12 
enrollments ranging from 17,000 to more than 


36,000. The free or reduced price lunch per- 
centage ranged from 41% to 64%. Students 
receiving special education services ranged 
from 10% to 18%, and were predominantly 
White in one district (62.1%) and predomi- 
nantly non-White in the other district (66%). 
The number of teacher participants varied 
across phases of the research, with 50 total 
teacher participants across all phases. In the 
development phase of the project, 17 teachers 
participated across sites; all worked in elemen- 
tary settings and identified as female. Partici- 
pants were primarily special education 
teachers, but also included four administrators, 
a reading specialist, and an English as a Sec- 
ond Language (ESL) teacher. Participants 
were predominantly White, and had an aver- 
age of 10 years teaching special education. For 
the feasibility phase of the project, 11 special 
education, ESL, and intervention teachers par- 
ticipated across sites. All were female and the 
majority self-identified as White (93%) and 
had earned at least a master’s degree (87%). 
Teachers had been in their current positions an 
average of 8 years. In the pilot study phase of 
the project, 22 teachers across sites who pro- 
vided direct support to students in Grades | to 
3 who had specific needs in beginning writing 
skills and who had or were at risk for disabili- 
ties were randomly assigned to treatment and 
control groups within school or within demo- 
graphically similar schools (n = 12 treatment 
and 10 control). The majority of participants 
were female (95%) and self-identified as 
White/European American (86%). Most were 
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special education teachers, with one ESL 
teacher in each group. More than 60% had at 
least a master’s degree and had taught in their 
current positions between 2 and 5 years on 
average, but had an average of 8 years of total 
experience. 


Assessments of Teacher and 
Student Learning 


In line with our theory of change, we collected 
data on important teacher outcomes that 
would lead to a greater understanding of their 
knowledge, skills, self-efficacy, and use of 
DBI with fidelity, including timely and appro- 
priate individualization of instruction. Each 
type of outcome is described in detail below. 


DBI Knowledge and Skills 


In the feasibility and pilot phases of the proj- 
ect, all teacher participants completed a DBI 
Knowledge and Skills pre—posttest. This 
assessment was developed and field-tested as 
part of the project. It included 40 multiple- 
choice questions that assessed teachers’ 
knowledge related to children’s writing devel- 
opment and writing instruction, the purpose of 
DBI, specific DBI steps, and skills related to 
administering, scoring, and using CBM writ- 
ing data to make instructional decisions, with 
number of items correct serving as the final 
score. Field-tests yielded internal consistency 
coefficients (Cronbach’s a) of .58 to .78. Indi- 
vidual items were revised based on item-level 
analyses; the final version was used for the 
pilot study. 


Teacher Efficacy and Writing 
Orientation 


In the pilot study phase of the project, teach- 
ers completed pre- and posttest surveys to 
explore the effect of DBI-TLC on teachers’ 
efficacy for teaching writing as well as their 
writing orientation. Critical to teachers’ suc- 
cessful use of student data are their knowl- 
edge of literacy development and practices, 
skill in selecting appropriate instruction to 
promote student learning (Cunningham et al., 


2004; Spear-Swerling, 2009), and self-effi- 
cacy. Self-efficacy relates to teachers’ “confi- 
dence that they can perform the actions that 
lead to student learning” (Graham et al., 
2001, p. 178). Teachers with strong self-effi- 
cacy are “better organized . . . more willing to 
try new ideas . . . more positive about teach- 
ing” (Graham et al., 2001, p. 178) and more 
likely to persist in helping struggling students 
(Allinder, 1994). In addition to assessing self- 
efficacy, one of our expert consultants on the 
project suggested that we examine teachers’ 
writing orientation, which influences how 
teachers teach and what they focus on while 
teaching (Graham, Harris, MacArthur, & 
Fink, 2002; Troia, Lin, Cohen, & Monroe, 
2011). Given strong evidence that explicit 
instruction is critical to support children who 
experience difficulties in learning to write 
(Berninger et al., 2008) and that the research- 
based intervention tools that we provided in 
this project focused on explicit instruction, 
we were interested in whether and how teach- 
ers’ writing orientation might change as a 
result of participating in DBI-TLC. 

Teacher efficacy was assessed using Gra- 
ham et al.’s (2001) modified version of Gib- 
son and Dembo’s (1984) Teacher Efficacy 
Scale (TES), which includes 16 items related 
to writing instruction at the elementary level. 
The teacher responds to statements such as, 
“When students’ writing performance 
improves, it is usually because I found better 
ways of teaching” (1 = strongly disagree; 6 = 
strongly agree). A factor analysis based on a 
national sample of teachers indicated two 
dimensions: personal teaching efficacy and 
general teaching efficacy, with Cronbach’s 
alphas of .84 and .69, respectively (Graham 
et al., 2001). 

Writing orientation was assessed using 
Graham et al.’s (2002) Writing Orientation 
Scale, a 13-item survey using a 6-point Lik- 
ert-type scale, with questions such as, “A 
good way to begin writing instruction is to 
have children copy good models of each par- 
ticular type of writing.” The questions align 
with the subscales Natural Writing, Correct 
Writing, and Explicit Instruction (Graham 
et al., 2002). Natural Writing includes an 
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emphasis on incidental and informal learning; 
Correct Writing places emphasis upon spell- 
ing, grammar, copying models, and using 
standard English; and Explicit Instruction 
includes teaching skills overtly and systemati- 
cally. 


CBM Reliability and Fidelity of DBI 
Components 


During the pilot study phase of the project, we 
periodically checked teachers’ reliability of 
CBM scoring, given that accurate scoring of 
student writing samples was critical to instruc- 
tional decision-making. Members of the 
research team collected students’ writing sam- 
ples from each teacher each month, rescored 
them, and compared the teachers’ scores with 
their own. Reliability was calculated as num- 
ber of agreements divided by agreements plus 
disagreements. 

Fidelity of DBI was assessed using the 
Accuracy of Implementation Rating Scales 
(AIRS), originally created by Fuchs, Deno, 
and Mirkin (1984) and modified for this proj- 
ect. The AIRS consists of three core DBI com- 
ponents: CBM, Writing Instruction, and 
Decision Making (DM). AIRS-CBM and 
Writing Instruction include detailed checklists 
of critical steps involved in assessment and 
intervention. AIRS-DM involves comparing 
teachers’ graphed data with a “decision log” 
in which they recorded information about 
instructional changes they made, including 
the date, type of change, and rationale for the 
change. Based on this information, we deter- 
mined whether the teachers’ decisions (a) 
were timely (1.e., they made a decision every 
six to eight data points, as prescribed in train- 
ing), (b) were appropriate (i.e., they used pre- 
scribed decision rules to determine whether to 
change instruction and how to change it), (c) 
indicated the change on the graph, and (d) 
noted the type of decision that was made. 
Fidelity was recorded as the number of items 
observed divided by the total number of appli- 
cable items. 

In addition, during the pilot study phase, 
we administered a brief survey to DBI-TLC 
and control teachers, asking them to estimate 


how frequently they made instructional deci- 
sions based on individual student progress. 
This information would allow a cursory 
examination of the extent to which DBI-TLC 
led to increased instructional decision-mak- 
ing. 


Student Outcomes 


Although not the primary focus of this article, 
it is worth noting that we also included mea- 
sures of student learning in the pilot study, 
because we hypothesized that student achieve- 
ment would be influenced by DBI-TLC 
implementation. Each participating teacher 
identified two to three students in their classes 
who needed intensive early writing interven- 
tion. Students were then screened and moni- 
tored weekly using CBM tasks for early 
writing, and were given a standardized writ- 
ing test, the Test of Early Written Language, 
3rd Edition (TEWL-III; Hresko, Herron, 
Peak, & Hicks, 2012), pre-post. More detail 
about these measures can be found in McMas- 
ter, Lembke, and Shin (2016). 


Development and 
Refinement of DBI-TLC 


A central aim of this project was to develop 
and iteratively revise and refine DBI-TLC. 
This iterative development process occurred 
across three phases, described in detail below. 


Development Phase 


The first phase involved development and 
refinement of a DBI manual and tools for 
implementing research-based early writing 
assessment, intervention, and data-based deci- 
sion-making, along with fidelity tools. We 
also developed learning modules to provide 
teachers with knowledge and skills needed to 
implement DBI. To develop these compo- 
nents, our research team reviewed the litera- 
ture in CBM for beginning writers, early 
writing intervention, and data-based decision- 
making. We gathered existing materials that 
could be used or modified for DBI “toolkits” 
for teachers. We drafted the manual, drawing 
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from similar existing manuals (e.g., Deno & 
Mirkin, 1977; Fuchs & Fuchs, 2007), and 
shared drafts with two national experts in 
CBM and DBI and with leaders from the par- 
ticipating districts. Their feedback on compre- 
hensiveness, organization, clarity, accuracy, 
and utility for teachers was incorporated into 
a revised version of the manual. 

CBM prompts and administration and 
scoring guides were drawn from previous 
research (e.g., Hampton & Lembke, 2016; 
Lembke, Deno, & Hall, 2003; McMaster, Du, 
& Pétursdottir, 2009; McMaster, Ritchey, & 
Lembke, 2011) and refined for use in this 
project. We developed graphing tools and a 
decision rubric to support teachers’ examina- 
tion, interpretation, and use of CBM data to 
inform instruction. In addition, results of a 
best-evidence synthesis (McMaster et al., in 
press) and input from a national expert in 
writing intervention were used to develop and 
refine research-based writing lessons and 
materials, along with guidance for teachers to 
analyze students’ strengths and weaknesses in 
writing and match interventions to their spe- 
cific needs. We also modified the AIRS (Fuchs 
et al., 1984) to assess fidelity of the assess- 
ment, instruction, and decision-making com- 
ponents of DBI. 

After developing the tools, we developed a 
series of learning modules, consisting of four 
daylong workshops that provided in-depth 
training in (a) the overall DBI process, (b) 
how to administer and score CBM for early 
writing and graph the data, (c) how to imple- 
ment and individualize early writing instruc- 
tion, and (d) how to use CBM data to intensify 
early writing intervention. Training included 
information about the writing process, includ- 
ing how students develop writing skills. The 
recommended research-based approaches we 
helped teachers to develop are based on theo- 
retical models of early writing development. 
Our particular focus was on the three key 
components of the Simple View of Writing 
(transcription, text generation, and self-regu- 
lation, Berninger & Amtmann, 2003). 
Throughout the training, we emphasized the 
alignment of these components with the writ- 
ing assessments and instructional approaches 


that were to be incorporated into the DBI pro- 
cess. 

We delivered the learning modules via a 
4-day Summer Institute to 17 teachers from 
the two participating districts. During the time 
in this Institute, teachers completed the DBI 
Knowledge and Skills assessment (pre- and 
posttest), learned all DBI procedures,received 
materials, and provided in-depth feedback 
regarding the learning modules and materials. 
Their feedback was elicited via semistruc- 
tured group discussions, written surveys, and 
a brief survey following the Institute. We also 
asked teachers to help generate ideas for the 
“collaborative support” component of DBI- 
TLC. 

All feedback was entered into a spread- 
sheet, categorized by themes, prioritized, and 
used to revise the tools and learning modules. 
Changes based on this feedback included cre- 
ating a “frequently asked questions” docu- 
ment and quick-start guides for each module 
to make start-up more feasible, changing the 
order of the learning modules to better reflect 
the steps of DBI, and streamlining the module 
content into three daylong workshops to fit a 
more realistic schedule. 

Based on teachers’ anticipated need for 
ongoing support, along with findings from a 
systematic literature review (Poch et al., in 
preparation) and input from a national PD 
expert, we also developed the collaborative 
support component, which we conceptualized 
as a cycle of coaching aligned to the learning 
modules. From the literature review (Poch 
et al., in preparation), we derived five princi- 
ples of coaching: Coaching should (a) be 
teacher oriented, (b) focus on building mas- 
tery, (c) be observable and measurable, (d) 
emphasize alignment with existing curricu- 
lum and instructional practices, and (e) be 
geared toward teachers’ sustained implemen- 
tation. The resulting collaborative support 
system included an opportunity for teachers to 
practice new content, share data, and discuss 
implementation issues with their coaches and 
peers. After each module, the teacher com- 
pleted a performance assessment to gauge 
mastery of DBI content. For example, after 
learning about CBM, the teacher scored CBM 
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samples, and was expected to meet a criterion 
of 85% scoring accuracy. The teacher’s coach 
then directly observed implementation of 
assessment and intervention, and provided 
feedback and support as needed via face-to- 
face (at least biweekly) and virtual (email, 
phone) sessions. Each coaching session fol- 
lowed a basic agenda with key steps that 
aligned with the coaching principles. Coaches 
were trained on the coaching process and 
principles at a “Coaches’ Institute.” 

Finally, we developed tools to examine 
fidelity of implementation of the “TLC” com- 
ponents of the package. These fidelity mea- 
sures included (a) Fidelity-T—an assessment 
of teachers’ accuracy in locating tools for the 
assessment, instruction, and decision-making 
components of DBI (all tools were provided 
in binders as well as on a Google drive); (b) 
Fidelity-L—detailed checklists of key points 
and activities to be covered in each of the 
learning modules; and (c) Fidelity-C—check- 
lists of key points to be covered during each 
coaching session. 


Feasibility Phase 


In the second phase of the project, we con- 
ducted a feasibility test of DBI-TLC. Eleven 
special education teachers and intervention 
specialists from the two sites received all DBI 
tools, participated in the learning modules, 
and received Coaching supports while they 
implemented DBI with students with inten- 
sive writing needs for 12 weeks. Teachers 
kept logs of time spent on various DBI activi- 
ties (e.g., collecting, scoring, and graphing 
CBM data; preparing instructional plans; 
making instructional decisions) for each stu- 
dent. Coaches kept logs of their interactions 
with teachers, including the frequency, focus, 
and outcomes of those interactions. At the end 
of the study, we invited teachers to attend a 
2-hour focus group at each site. We developed 
a semistructured protocol to elicit teachers’ 
input on the feasibility of the entire DBI pro- 
cess, the learning curve, the cycle of imple- 
mentation (including learning modules and 
coaching), and the DBI tools. Teachers also 
completed an anonymous survey about their 


satisfaction with the Coaching components of 
DBI-TLC, a 12-item, researcher-developed 
instrument where teachers were asked six 
open-ended questions about the strengths and 
weaknesses of coaching. This coaching sur- 
vey was used to iteratively develop and revise 
our coaching methods. 

Data from teachers’ and coaches’ logs, the 
focus group, and the coaching survey were 
entered into a spreadsheet. Then, two research- 
ers developed a coding scheme to capture all 
feedback from these data sources. We began 
by independently identifying themes to cate- 
gorize the feedback, then compared notes and 
came to agreement through an iterative pro- 
cess of generating and comparing themes 
until we agreed on all codes. This process 
resulted in three “levels” of coding: Level 1 
distinguished among what teachers perceived 
to be actual facilitators (features of DBI-TLC 
that teachers perceived fostered implementa- 
tion of DBI), suggested facilitators (features 
that teachers suggested would facilitate imple- 
mentation if incorporated into DBI-TLC), and 
barriers/challenges (things that teachers per- 
ceived stood in the way or made DBI imple- 
mentation difficult). Level 2 indicated whether 
the perceived facilitators and barriers were 
related to DBI, tools, learning modules, or 
collaborative support components of DBI- 
TLC. 

Level 3 entailed more specific, descriptive 
categories. For example, teachers cited a spe- 
cific actual facilitator (Level 1) of DBI (Level 
2) to be its alignment (Level 3) with existing 
instructional programming or with students’ 
needs. A specific suggested facilitator (Level 
1) of DBI tools (Level 2) was to improve their 
usability (Level 3) by making more ready- 
made materials. A specific barrier/challenge 
(Level 1) to coaching (Level 2) included 
external conflicts (Level 3) such as limited 
time to schedule coaching sessions. Complete 
definitions and examples of codes are reported 
in Poch, McMaster, and Lembke (2016). 

Results of this analysis were used to both 
evaluate the feasibility of DBI-TLC and to 
inform further revisions (for a full description 
of analysis and results, see Poch et al., 
2016). With respect to feasibility, teachers 
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highlighted several key facilitators, including 
that the step-by-step process made DBI less 
overwhelming than other new programs, that 
the assessment and intervention procedures 
were well aligned with and easy to incorpo- 
rate into existing instructional programs, that 
they found the data to be valuable, and that 
they were likely to continue using DBI in the 
future. From this feedback, we concluded that 
the overall DBI-TLC components and proce- 
dures were generally feasible to implement in 
school settings. 

To further revise DBI-TLC, we attended 
closely to suggested facilitators and barriers/ 
challenges that teachers identified. Most of 
this feedback related to challenges with orga- 
nizing and managing DBI tools. Thus, we 
heeded teachers’ suggestions to make inter- 
vention materials more accessible, organized, 
and easy to use. For example, we had pro- 
vided all materials electronically via a shared 
Google drive, but many teachers had diffi- 
culty navigating this drive. They preferred 
that we printed and organized the materials in 
large three-ring binders. Teachers also sug- 
gested that the initial learning modules occur 
before the school year begins so that they 
could begin administering CBM and deliver- 
ing interventions right away. Also, many 
teachers found the learning curve to be steep 
at first but to level off over time, and thus 
desired more coaching initially with less 
coaching after DBI was up and running. This 
feedback was used to adjust the schedule of 
delivering learning modules and coaching 
activities. 


Pilot Study 


In the last phase of the project, we evaluated 
the promise of DBI-TLC to improve both 
teacher and student outcomes in a small, ran- 
domized pilot study using a pre- and posttest 
control group design. Participants were from 
two Midwestern districts, one large urban and 
one midsized city with diverse student popu- 
lations, and included 20 special education 
teachers primarily serving children in Grades 
1 to 3 (randomly assigned to DBI or Control) 
and 57 children with a range of mild to moder- 


ate disabilities and significant early writing 
needs. 

As part of this study, we collected fidelity 
data on teachers’ CBM administration, inter- 
vention implementation, and decision-mak- 
ing, as well as fidelity of teachers’ access of 
DBI tools, our own delivery of the learning 
modules, and coaching. These fidelity obser- 
vations revealed a couple of issues: (a) fidelity 
tools, particularly for intervention, did not 
necessarily differentiate among DBI and con- 
trol teachers, nor did they capture the quality 
of implementation; and (b) teachers did not 
always make timely or appropriate decisions 
based on data. These findings led us to con- 
sider ways to improve both our fidelity tools 
(to make them more sensitive to the quality of 
DBI implementation) and our training and 
coaching activities (to improve teachers’ use 
of data to make instructional decisions). We 
discuss these issues in more depth in the 
“Summary and Implications” section. 


Key Findings 
Knowledge and Skills 


At both the feasibility and pilot phases of the 
project, teachers’ DBI knowledge and skills 
appeared to improve following participation 
in DBI-TLC activities. During the Feasibility 
Study, teachers’ mean pretest performance 
was 28.45 items correct (SD = 4.68). Twelve 
weeks later, after receiving the learning mod- 
ules and collaborative support, mean posttest 
performance was 36.27 (SD = 2.69). A paired- 
samples ¢ test revealed statistically significant 
growth from pre- to posttest, #(10) =—6.659, p 
< .001, d= 1.90. 

Whereas findings from the feasibility 
phase were encouraging, the lack of a control 
group precluded any causal inference we 
could make about the effect of DBI-TLC on 
teachers’ knowledge and skills. The random- 
ized control trial conducted during the pilot 
phase, however, enabled us to examine 
whether gains made over time could be attrib- 
uted to DBI-TLC. Using the pre- and posttest 
data collected during the Pilot Study, we ran a 
repeated-measures analysis of variance 
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Table |. Pilot Study Teacher Outcomes by 
Condition. 


DBI-TLC Control 
(n= 11) (n= 9) 
M SD M SD 
Knowledge and Skills 
Pretest 25.55 495 25.67 2.78 
Posttest 32.32 2.26 23.50 3.52 


Teacher Efficacy 
General Efficacy 
Pretest 4.18 0.88 4.50 0.80 
Posttest 454 0.75 4.51 0.67 
Personal Efficacy 
Pretest 454 0.35 446 0.35 
Posttest 4.94 051 446 0.50 
Writing Orientation 
Correct Writing 
Pretest 3.03 1.05 2.50 3.03 
Posttest 3.23. 0.72 3.00 0.87 
Explicit Instruction 
Pretest 5.00 0.65 4.56 0.64 
Posttest 5.77 0.25 5.06 0.51 
Natural Learning 
Pretest 4.37 0.56 4.24 0.55 
Posttest 3.72 0.69 4.26 0.57 


DBI-TLC: Data-Based Instruction-Tools, Learning, and 
Collaborative Support. 


(RM-ANOVA) with time (pre- vs. posttest) as 
the within-subjects factor and condition (DBI- 
TLC vs. control) as the between-subjects fac- 
tor (see Table 1 for means and standard 
deviations). This analysis revealed a statisti- 
cally significant time by condition interaction, 
FC, 16) = 25.99, p< .001. Whereas DBI-TLC 
and control teachers performed similarly at 
pretest (d = —.03), DBI-TLC teachers outper- 
formed controls at posttest on the Knowledge 
and Skills measure, with a large effect size (d 
= 3.05). Thus, it appears that DBI-TLC did, in 
fact, explain changes in teachers’ DBI knowl- 
edge and skills. 


Teacher Efficacy and Writing 
Orientation 
Teacher efficacy and writing orientation were 


assessed only during the pilot test phase of the 
project. Means and standard deviations for 


each subscale can be found in Table 1. Linear 
regression, in which efficacy posttest scores 
were regressed on condition, pretest scores, 
and an interaction between pretest scores and 
condition, revealed few differences between 
treatment and control teachers. Most teachers 
had high self-efficacy across conditions and 
times as well as mixed writing orientations 
(most teachers associated with more than one 
writing orientation). The explicit instruction 
subscale of the Writing Orientation Scale 
(Graham et al., 2002) was the only subscale in 
which a main effect of condition was signifi- 
cant when controlling for pretest score (R* = 
.63, p < .001); there were no significant inter- 
actions among the predictor variables. Teach- 
ers in the treatment condition scored an 
average of .56 (p = .0034) higher than the 
control group on the posttest. 


CBM Scoring Reliability and Fidelity 
of DBI Implementation 


Periodic reliability checks indicated that 
teachers scored their students’ CBM samples 
reliably (mean interrater agreement between 
teachers and scoring experts = 95%). Fidelity 
observations indicated that, on average, teach- 
ers administered CBM with 83% accuracy 
(range = 69%-100%), intervention with 79% 
accuracy (range = 40%-94%), and overall 
decision-making fidelity was 52% accuracy 
(range = 0%-88%). Furthermore, most DBI 
teachers (n = 10) reported implementing mul- 
tiple instructional changes during the study 
period, whereas only one control teacher 
reported doing so. 


Student Outcomes 


As we mentioned earlier, student outcomes 
are not a primary focus of this article; these 
outcomes are described in detail in McMaster 
et al. (2016). Briefly, a series of hierarchical 
linear models (HLM; Raudenbush & Bryk, 
2002), with students nested within teacher and 
controlling for students’ pretest performance, 
indicated a pattern of higher mean perfor- 
mance of DBI students compared with con- 
trols on proximal CBM tasks, with small to 
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moderate effect sizes of d = .23 to .40, though 
these differences were not statistically signifi- 
cant. There were no reliable differences on the 
TEWL-III (Hresko et al., 2012), and effect 
sizes were small (d = .10-.18). Note that, 
given that the focus of this project was on 
iterative development and the aim was to pro- 
vide preliminary evidence of feasibility and 
promise, the pilot study was underpowered. 
We are encouraged by the initial promising 
results and are hopeful that future efficacy 
work conducted with larger samples of stu- 
dents will produce stronger results. 


Summary and Implications 


In this project, we set out to develop and eval- 
uate the feasibility and promise of DBI-TLC, 
a PD system designed to support teachers’ 
implementation of DBI to address children’s 
intensive early writing needs. In line with 
Desimone’s (2009) framework for PD 
research, our work was guided by a theory of 
change that posited that DBI-TLC would 
serve to improve teachers’ knowledge, skills, 
and self-efficacy related to DBI and writing 
instruction, which in turn would support their 
implementation of DBI with fidelity, includ- 
ing timely and appropriate instructional 
changes, which would ultimately lead to 
improved student outcomes. 

Findings from this project support compo- 
nents of this theory of change to varying 
degrees. First, evidence from the DBI Knowl- 
edge and Skills assessment indicated that 
DBI-TLC did, indeed, lead to improved 
teacher knowledge and skills. This result 
likely reflects the focus on content prescribed 
by Desimone’s (2009) framework. Although 
teachers’ overall efficacy for writing instruc- 
tion did not appear to change, we did observe 
that those who participated in DBI-TLC 
shifted to a more explicit writing orientation 
than did their control counterparts. We also 
observed that teachers were able to administer 
CBM with reasonable fidelity and score stu- 
dent samples reliably, and that they imple- 
mented instruction with some degree of 
fidelity (though instructional fidelity varied 
quite widely). These results are in line with 


Desimone’s (2009) emphasis on active learn- 
ing opportunities over an extended period of 
time. Less evident was teachers’ fidelity of 
decision-making, with most teachers making 
timely and appropriate decisions less than half 
of the time. Finally, whereas children’s writ- 
ing outcomes showed promising patterns on 
proximal measures as a result of teachers’ 
DBI implementation, these patterns were not 
statistically reliable, nor did they generalize to 
more distal measures of writing. 


Implications for Research 


Overall, we are encouraged that DBI-TLC 
improved teachers’ knowledge and skills, 
because knowledge and skills are likely nec- 
essary for changing teacher practice (e.g., 
Cunningham et al., 2004; Spear-Swerling, 
2009). We are also encouraged that DBI-TLC 
teachers shifted to a more explicit writing ori- 
entation, as research indicates the importance 
of explicit instruction for fostering children’s 
early writing development (e.g., Berninger 
et al., 2008). In addition, research is needed to 
shed light on whether these factors serve as 
mediators of improved student outcomes; we 
were unable to explore such relations given 
our small sample, but intend to as part of 
future work. 

Furthermore, although teacher efficacy did 
not appear to be affected by DBI-TLC, we are 
not necessarily discouraged by this outcome, 
because teacher efficacy was relatively high at 
first. This finding is in line with previous stud- 
ies (e.g., Graham et al., 2001; Graham et al., 
2002; Ritchey, Coker, & Jackson, 2015; Troia 
et al., 2011), and might reflect that teachers 
were study volunteers (a prerequisite of par- 
ticipating in this project)—Teachers moti- 
vated to participate in the research might 
already have a relatively strong sense of their 
capacity to effect improved student outcomes. 
Furthermore, research is needed to determine 
the extent to which teachers’ efficacy affects 
their instructional practices and, by extension, 
student outcomes (Graham et al. 2002; 
Ritchey et al., 2015; Troia et al., 2011). A 
related, important question for future DBI 
research is whether efficacy of a wider range 
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of teachers who participate in PD such as 
DBI-TLC is altered, particularly for those 
who begin with relatively low self-efficacy. It 
might also be useful to determine whether lev- 
els of teacher knowledge, skill, self-efficacy, 
and writing orientation can be used to indi- 
vidualize teacher PD and support for DBI. 

We are somewhat encouraged that teachers 
were able to implement aspects of DBI with 
reasonable fidelity (assessment and interven- 
tion components were implemented, on aver- 
age, with close to 80% accuracy), though 
many questions remain. For example, it is not 
clear what level of fidelity is sufficient to lead 
to improved student outcomes. Also unclear is 
how best to assess fidelity when teachers are 
encouraged to individualize instruction (and 
thus possibly stray from prescribed interven- 
tion procedures). In our project, we discov- 
ered that it was particularly challenging to 
create a tool that was sufficiently detailed to 
capture critical instructional elements, but 
also sufficiently generic to allow for teachers’ 
instructional adjustments. In addition, we 
found our measures to be inadequate in terms 
of capturing instructional quality, including 
the extent to which teachers engaged students 
in meaningful ways and were appropriately 
responsive to students when engagement 
waned. Thus, we are currently exploring more 
in-depth ways to assess fidelity and instruc- 
tional quality, including the fidelity of teach- 
ers’ intensification of instruction, and the 
extent to which teachers foster and reinforce 
student engagement during instruction. 

We are somewhat discouraged that teach- 
ers largely failed to make what we operation- 
alized as timely and appropriate instructional 
changes to individualize instruction for stu- 
dents. We do not necessarily view this out- 
come as a measurement problem (as we 
described for instructional fidelity), but pos- 
sibly as an indicator of limitations to the 
learning and collaborative support opportuni- 
ties provided through DBI-TLC that shored 
up teachers’ decision-making capacity. Per- 
haps Desimone’s (2009) notion of coherence, 
ensuring that DBI components align with 
theory and core academic standards, and are 
integrated into existing instructional routines, 


was limited as DBI was not something that 
most of our teachers were used to doing. Fur- 
thermore, work is needed to determine how 
best to teach teachers to correctly interpret 
data, as well as to act upon the data in a timely 
way. We are revisiting our learning modules 
and coaching procedures to determine where 
there might be areas where we need to shore 
up supports for teachers in their use of data. A 
recent special issue of Learning Disabilities 
Research & Practice (February 2017, Volume 
32, Issue 1) on teachers’ use of data for 
instructional decision-making also provides 
valuable insights that we can draw from in 
this regard. 


Implications for Practice 


Although we view our findings as preliminary 
given the developmental nature of the above- 
described work, we believe that there are sev- 
eral implications for teacher development that 
can be applied to both preservice and in-ser- 
vice training. First, there is a reported wide- 
spread lack of attention to writing assessment 
and instruction in teacher preparation and PD 
programs (Cutler & Graham, 2008). We 
believe we can fill this gap with the research- 
based tools and learning modules that we have 
developed as part of this project, and that use 
of these products has great promise to improve 
teacher knowledge and skills related to DBI 
and early writing. We have already begun to 
do so, by incorporating these resources in our 
teacher preparation programs and making 
them available through http://arc.missouri. 
edu/dbi_early.aspx. Second, providing tools 
and learning opportunities is necessary, but 
not likely sufficient for changing teacher prac- 
tices. Many teachers are likely to require 
ongoing supports to successfully adopt and 
implement DBI in early writing with fidelity. 
Third, particular attention is needed for 
supporting teachers’ use of data for instruc- 
tional decision-making. This component of 
DBI is essential, yet has remained one of the 
most difficult skills for teachers to learn and 
implement (Espin et al., in press; Stecker 
et al., 2005). It may be that data-based deci- 
sion-making requires a level of expertise that 
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can only be developed through more exten- 
sive and intensive PD, support, and experi- 
ence than what we provided through 
DBI-TLC. Throughout our project, we uti- 
lized Desimone’s (2009) feature of effective 
PD, collective participation, ensuring that 
teachers have frequent opportunities to col- 
laborate with coaches and peers. Outside of 
the project, however, this level of support may 
not always be available. Furthermore, research 
is needed to determine how best to achieve 
sustainable results, including an examination 
of the role of instructional leaders. Research- 
ers and practitioners should continue to work 
together to find ways to develop this type of 
instructional expertise, so that children with 
the most intensive early writing needs can 
receive the maximum benefit possible from 
individualized writing instruction. 
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